Method for augmented reality display, method for determining pose information, and apparatuses

ABSTRACT

A method for an image capturing device includes: acquiring a first image of a target object; extracting coordinate information of a first 2D key point corresponding to the target object; acquiring a second image of the target object; acquiring coordinate information of a second 2D key point in the second image according to the coordinate information of the first 2D key point, the first 2D key point and the second 2D key point being the same key point in the target object; acquiring coordinate information of a 3D key point corresponding to the target object, the 3D key point being a key point obtained after a triangulation operation; and determining camera posture information of the second image according to the coordinate information of the first 2D key point, the coordinate information of the second 2D key point, the coordinate information of the 3D key point, and a 3D-2D projection.

RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/778,259, filed on Jan. 31, 2020. U.S. patentapplication Ser. No. 16/778,259 is a continuation application of PCTPatent Application No. PCT/CN2018/097494, filed on Jul. 27, 2018, whichclaims priority to Chinese Patent Application No. 201710643376.8,entitled “METHOD FOR AUGMENTED REALITY DISPLAY, METHOD FOR DETERMININGPOSE INFORMATION, AND APPARATUSES” filed with the Chinese Patent Officeon Jul. 31, 2017, content of all of which is incorporated by referencein its entirety.

FIELD OF THE TECHNOLOGY

Embodiments of the present disclosure relate to the field ofsimultaneous localization and mapping (SLAM) and, in particular, to amethod for determining information of an image capturing device andrelated apparatuses.

BACKGROUND

In the field of SLAM, when entering a strange environment, a device (forexample, a robot or a virtual reality (VR) device) needs to preciselyestablish a correspondence between time and space.

At present, in a SLAM system, during the tracking of a camera,calculated three-dimensional (3D) points are usually projected on a newframe of two-dimensional (2D) image, and camera posture information ofthe camera is estimated. To be specific, a manner of 3D-2Dcorrespondences is used to perform tracking.

FIG. 1 is a schematic diagram of acquiring 3D points based on 3D-2Dcorrespondences in the related art. Referring to FIG. 1, during theestimation of camera posture information of a t^(th) frame of image, a(t−1)^(th) frame of image and a (t−2)^(th) frame of image need to beused to perform triangulation on key points, 3D positions (hollow starsin FIG. 1) of the key points are obtained, and the 3D positions of thekey points are then used to estimate the camera posture information ofthe t^(th) frame of image. In the case of fast movement from the(t−2)^(th) frame of image to the t^(th) frame of image, hollow starsonly account for a small part of the t^(th) frame of image. When keypoints in such a small part are used to estimate the camera postureinformation of the entire t^(th) frame of image, a relatively largeerror may occur. In addition, a tracking loss tends to occur because ofan excessively small quantity of key points in tracking.

The disclosed methods and systems are directed to solve one or moreproblems set forth above and other problems.

SUMMARY

Embodiments of the present disclosure provide a method for augmentedreality (AR) display, a method for determining posture information, andapparatuses, to obtain more key points during the estimation of cameraposture information, so as to reduce errors in camera postureinformation, and to avoid tracking loss.

According to one aspect of the present disclosure, a method for ARdisplay is provided for an image capturing device. The method includesacquiring a first image of a target object; extracting coordinateinformation of a first two-dimensional (2D) key point corresponding tothe target object from the first image; acquiring a second image of thetarget object; and acquiring coordinate information of a second 2D keypoint in the second image according to the coordinate information of thefirst 2D key point. The first 2D key point and the second 2D key pointare the same key point in the target object. The method also includesacquiring coordinate information of a three-dimensional (3D) key pointcorresponding to the target object, the 3D key point being a key pointobtained after a triangulation operation; and determining camera postureinformation of the second image according to the coordinate informationof the first 2D key point, the coordinate information of the second 2Dkey point, the coordinate information of the 3D key point and a 3D-2Dprojection, the camera posture information being used for determining aposition of the target object in the second image.

According to another aspect of the present disclosure, an imagecapturing device is provided. The image capturing device includes amemory being configured to store a computer program; a bus system; and aprocessor connected to the memory via the bus system. The processor isconfigured to execute the computer program in the memory to perform:acquiring a first image of a target object; extracting coordinateinformation of a first two-dimensional (2D) key point corresponding tothe target object from the first image; acquiring a second image of thetarget object; acquiring coordinate information of a second 2D key pointin the second image according to the coordinate information of the first2D key point, the first 2D key point and the second 2D key point beingthe same key point in the target object; acquiring coordinateinformation of a three-dimensional (3D) key point corresponding to thetarget object, the 3D key point being a key point obtained after atriangulation operation; and determining camera posture information ofthe second image according to the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key point, thecoordinate information of the 3D key point, and a 3D-2D projection, thecamera posture information being used for determining a position of thetarget object in the second image.

According to one aspect of the present disclosure, a non-transitorycomputer-readable storage medium is provided. The non-transitorycomputer-readable storage medium stores computer program instructionsexecutable by at least one processor to perform: acquiring a first imageof a target object; extracting coordinate information of a firsttwo-dimensional (2D) key point corresponding to the target object fromthe first image; acquiring a second image of the target object;acquiring coordinate information of a second 2D key point in the secondimage according to the coordinate information of the first 2D key point,the first 2D key point and the second 2D key point being the same keypoint in the target object; acquiring coordinate information of athree-dimensional (3D) key point corresponding to the target object, the3D key point being a key point obtained after a triangulation operation;and determining camera posture information of the second image accordingto the coordinate information of the first 2D key point, the coordinateinformation of the second 2D key point, the coordinate information ofthe 3D key point, and a 3D-2D projection, the camera posture informationbeing used for determining a position of the target object in the secondimage.

Other aspects of the present disclosure can be understood by thoseskilled in the art in light of the description, the claims, and thedrawings of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of acquiring 3D points based on 3D-2Dcorrespondences;

FIG. 2 is a schematic diagram of acquiring 3D points based on 3D-2Dcorrespondences according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of an interface that presents a targetobject according to camera posture information according to anembodiment of the present disclosure;

FIG. 4 is a schematic diagram of a method for determining information ofan image capturing device according to an embodiment of the presentdisclosure;

FIG. 5 is a schematic diagram of determining an observation angleaccording to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of generating a tracking record accordingto an embodiment of the present disclosure;

FIG. 7 is a schematic flowchart of determining camera postureinformation in an application scenario according to an embodiment of thepresent disclosure;

FIG. 8 is a schematic diagram of an image capturing device according toan embodiment of the present disclosure;

FIG. 9 is a schematic diagram of an image capturing device according toanother embodiment of the present disclosure;

FIG. 10 is a schematic diagram of an image capturing device according toanother embodiment of the present disclosure; and

FIG. 11 is a schematic structural diagram of an electronic deviceaccording to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide a method for AR display, amethod for determining posture information, and apparatuses, to obtainmore key points during the estimation of camera posture information of asecond image, so that errors in camera posture information are reduced,and a tracking loss because of an excessively small quantity of keypoints in tracking can be avoided.

In the specification, claims, and accompanying drawings of the presentdisclosure, the terms “first”, “second”, “third”, “fourth”, and the like(if existing) are intended to distinguish between similar objects ratherthan describe a specific sequence or a precedence order. It is to beunderstood that data used in this way is interchangeable in a suitablecase, so that the embodiments of the present disclosure described hereincan be implemented in a sequence in addition to the sequence shown ordescribed herein. Moreover, the terms “include”, “contain” and any othervariants mean to cover the non-exclusive inclusion. For example, aprocess, method, system, product or device that includes a list of stepsor units is not necessarily limited to those steps or units clearlylisted, but may include other steps or units not expressly listed orinherent to such a process, method, product or device.

It is to be understood that this solution is mainly applied to a SLAMsystem, and is used for a device to locate itself based on locationestimation and a map during movement and also build an incremental mapbased on its positioning, thereby implementing autonomous positioningand navigation of the device. This solution may be specifically appliedto VR products, AR products, and Visual Robot Navigation (VRN) products,and the like. Specific products are not limited herein.

VR essentially provides immersive experiences to enable a user toexperience a completely different virtual world. SLAM is the perceptionand comprehension of the real world. If a VR product requires SLAM, thevirtual world and the real world can be combined. In commerciallyavailable VR products, a six-axis gyroscope is used to track therotation of the head of a user but cannot track the displacement of theuser. However, SLAM can resolve the tracking in six degrees of freedom.

AR is essentially perfect integration of virtual elements in reality.Compared with VR products, AR products have much more complexalgorithms, software, and hardware, and the mass production of ARproducts are much more difficult. It is imperative that AR is providedwith a SLAM system.

The embodiments of the present disclosure provide a camera trackingmethod combining 3D-2D correspondences and 2D-2D correspondences, inwhich more stable 2D-2D correspondences are used to assist in trackingwith 3D-2D correspondences, thereby improving the tracking stability ofthe SLAM system. FIG. 2 is a schematic diagram of acquiring 3D pointsbased on 3D-2D correspondences according to an embodiment of the presentdisclosure. Referring to FIG. 2, during the estimation of camera postureinformation of a t^(th) frame of image, a (t−1)^(th) frame of image anda (t−2)^(th) frame of image need to be used to perform triangulation onkey points, 3D positions (hollow stars in FIG. 2) of the key points areobtained, the t^(th) frame of image and the (t−1)^(th) frame of imagealso need to be used to obtain 2D positions (solid stars in FIG. 2) ofthe key points, and the 2D positions and the 3D positions of the keypoints are then used to estimate the camera posture information of thet^(th) frame of image.

For ease of understanding, target objects presented in the embodimentsof the present disclosure may be described below by using a specificapplication scenario. FIG. 3 is a schematic diagram of an interface thatpresents a target object according to camera posture informationaccording to an embodiment of the present disclosure. Referring to FIG.3, the image on the left is the picture before movement, and the imageon the right is the picture after movement. A user may see on the screenof a mobile phone a virtual object (for example, the pig in FIG. 3)moving at a fixed position (for example, on a table) in a real 3D scene,and the virtual object does not move as the user's mobile phone moves.

The embodiments of the present disclosure may be applied to an ARsystem. A method for AR display provided in the embodiments of thepresent disclosure is described below. The method for AR displayincludes the followings.

First, an image capturing device acquires a target object from differentphotographing angles in the same real scene. The target object may be abook, a tree, a television set or a can of cola, etc. It is intended tointegrate a virtual object with the target object. The virtual objectmay be a cartoon character designed beforehand by a designer or adoodle, without limitation.

After the target object is determined, the image capturing deviceextracts coordinate information of a first 2D key point corresponding tothe target object from a first image. Next, the image capturing devicecontinues to acquire a second image. It may be understood that thesecond image and the first image come from the same target object. Theimage capturing device may acquire coordinate information of a second 2Dkey point in the second image according to the coordinate information ofthe first 2D key point. The first 2D key point and the second 2D keypoint are the same key point in the target object.

The image capturing device needs to acquire a 3D key point, and maydetermine coordinate information of the 3D key point. The coordinateinformation of the 3D key point mainly includes coordinate informationin six dimensions, that is, displacements in three directions and atthree angles. The 3D key point is a key point on which triangulationprocessing has been successfully performed in previous frames of image.The image capturing device determines camera posture information of thesecond image according to the coordinate information of the first 2D keypoint, the coordinate information of the second 2D key point, and thecoordinate information of the 3D key point. The camera postureinformation is used for determining a position of the target object inthe second image. The camera posture information may representphotographing space information of the second image.

The image capturing device acquires additional image information. Theadditional image information is an image in a photographed real scene,for example, a swimming pool, a garden or a high school playground.Finally, the image capturing device combines the second image, thecamera posture information of the second image, and the additional imageinformation into an augmented image. For example, the additional imageinformation is an image of a high school playground. The second imageincludes a tree. It is determined according to the camera postureinformation of the second image that there is an animated bird in thetree. In this case, the augmented image shows the foregoing content. Theaugmented image is used for displaying AR content.

Several application scenarios in which the method for AR displayprovided in the embodiments of the present disclosure is applied aredescribed below.

1. AR Map

The method for AR display may be used to present a navigationapplication in which a pedestrian can easily follow a route. In theapplication, an arrow and a route are directly added to a realenvironment, to provide a highly intuitive navigation route. Inaddition, the method for AR display may also be used to recognize locallandmarks and provide useful information about such buildings, forexample, names of places and distances between buildings. This is notlimited.

The image capturing device can recognize places or objects according toscene-sensitive information software, and joins digital information andreal-world scenes. The scene-sensitive information software includes anapplication that can readily acquire existing static data on theInternet according to a specific scene. Such software may be run on asmart terminal. A user may use the viewing angle of a camera on thesmart terminal to see surrounding digital information. Such digitalinformation may be nearby places of interest, for example, a museum, astore, a restaurant or a walking route to a next bus stop. The softwareimplements image recognition by using the Global Positioning System(GPS), Wireless Fidelity (Wi-Fi), and 3D modeling, to implement a userpositioning function.

2. AR Language Translation

Language translation is one of the most promising fields in ARapplications. The method for AR display provided in the embodiments ofthe present disclosure may be applied to a smart terminal. The smartterminal may be used to synchronously translate text into text inanother language. After opening an application, a user only needs topoint the image capturing device at text in a foreign language. Theimage capturing device translates the information into information inthe user's native language. The translated text has the same font and isprinted on the same wall just like the original text.

3. AR Measurement Tool

Smart terminals oftentimes can nearly replace personal computers (PCs),but still cannot replace some tools such as a ruler or a trianglerequired in life. However, with AR technology, smart terminals may alsobe used as measurement tools. For example, by means of the method for ARdisplay provided in the embodiments of the present disclosure, a virtualruler may be integrated with the reality. A smart terminal is alreadyprovided with functions of a camera, a processor, a motion sensor, andthe like. These are hardware conditions for virtual measurement tools.In addition, AR display technology is used to display two floatingpoints on an object to be measured in reality. Data can therefore beobtained.

A virtual magnetic measurement tool may be consistent with a tapemeasure in reality. A user only needs to hold the smart terminal,confirm a starting point of a virtual magnetic measurement ruler on thescreen, move the smart terminal horizontally, and confirm the end pointto obtain data immediately. Apart from a distance, AR display technologyalso makes it very easy to measure a space. For example, the user mayhold the smart terminal and move the smart terminal along the edges of aroom without needing to record the sizes of sides. After the smartterminal has been moved around a space to be measured, the system givescalculated data. The area of a space with any irregular shape can beeasily calculated.

4. AR Furniture

By means of the AR display technology provided in the embodiments of thepresent disclosure, an application may further be designed. A consumermay tap and choose a digital furniture on a smart terminal and “place”it in the consumer's living room, so that it is more convenientlychecked whether the size, style, and color of the furniture are suitablefor a position. The application also allows a user to adjust the sizeand color of each part.

The effect of placing a furniture in a room is usually visualized beforepurchase. In the foregoing manner, it is only necessary to hold a smartterminal and point the camera at each corner of a room to pick a mostsuitable furniture. This also gives rise to a new sales approach infurniture retail. Sellers no longer need to bear the costs of large-areabrick-and-mortar shops, and consumers can save the time and labor ofbrowsing furniture shops.

Similarly, in some scenarios of fittings, according to the AR displaytechnology provided in the embodiments of the present disclosure, a moreuser-friendly AR fitting experience can be provided. The size of acustomer and the chosen clothes can be recognized. Therefore, it is nolonger necessary for the customer to try on clothes with differentcolors. The customer only needs to put on a piece of clothing, stand infront of a mirror, and choose other colors according to prompts on atouchscreen, so that the image of the customer in the piece of clothingwith a different color may be presented.

5. AR Teaching

When AR is used for dance teaching, the limitations in space and timecan be removed for students and teachers. By means of the AR displaytechnology described in the embodiments of the present disclosure,virtual “steps” may be cleverly applied to the reality. A student maylearn dance steps by following virtual footprints. An application caneven slow down according to a student's requirement, and enables astudent to learn dancing with a dancing partner.

Similarly, apart from dance teaching, the AR display technology providedin the embodiments of the present disclosure may also be applied tomartial art teaching, basketball teaching, swimming teaching, and thelike. This is an example herein, and does not constitute a limitation tothe embodiments of the present disclosure.

6. AR Logistics

Next, the logistics industry is used as an example to specificallydescribe the potential influence of the AR display technology. Althoughthe application of AR in the logistics industry is still in a relativelyearly stage, AR can provide huge benefits. For example, AR enables alogistics provider to acquire information rapidly anytime anywhere. Thisis vital for precise planning and meticulous operation of tasks such asdelivery and loading optimization, and also forms a solid basis forcustomer services with higher quality. Some instances are conceived forthe application of AR in the logistics industry. These instances includethe following four types, namely, warehouse operations, transportationoptimization, last-kilometer distribution, and enhanced value-addedservices.

Warehouse operations are the most promising application field of AR inlogistics. These operations take up approximately 20% of the totallogistics cost. Picking tasks take up 55% to 65% of the total costs ofwarehouse operations. AR may start with improving a picking procedure,thereby greatly reducing the operation cost. AR also facilitates thetraining of new warehouse employees and temporary employees, andprovides reference for warehouse planning. The picking procedure can beoptimized by using visual picking.

In logistics, the most practical AR solution is a system that canoptimize a picking procedure. Most of the warehouses in developedcountries still use a paper-based picking approach. However, anypaper-based approach is inefficient and error-prone. In addition,picking is usually performed by temporary workers. These people oftenrequire training at some costs to ensure efficient picking with as fewmistakes as possible.

A visual picking system is currently in a final on-site test stage. Thesystem includes a mobile AR apparatus such as a head mount display(HMD), a camera, a wearable PC, and a battery module that at least lastsone shift. The function of visual picking software in the systemincludes real-time object recognition, barcode reading, indoornavigation, and seamless information integration with a warehousemanagement system (WMS). The greatest benefit of visual picking is thata warehouse worker can obtain intuitive digital information supportwithout using hands during manual picking. By means of such a system,warehouse workers can see digital picking lists in their views and cansee the optimal route with the help of an indoor navigation function,thereby reducing the movement time through effective route planning.Image recognition software in the system can automatically read abarcode to determine whether a warehouse worker reaches a correctposition, and directs the warehouse worker to rapidly locate an objectto be picked from a shelf. Next, the warehouse worker can scan theobject to synchronously register this procedure in the WMS to implementreal-time stock update. In addition, systems of such types can reducethe time required for training new employees, and can remove languagebarriers that inadequately literate warehouse workers may encounter.

On-site tests of these AR systems have proven that such systemscontribute greatly to improving the efficiency of warehouse operations.For example, a continuous picking verification function can reducemistakes by 40%. It may be understood that the AR systems herein use themethod for AR display provided in the embodiments of the presentdisclosure.

During warehouse planning, AR may be probably beneficial to a warehouseplanning procedure. Warehouses nowadays are no longer only nodes forstorage and distribution, and gradually provide more and morevalue-added services, including the assembly to labeling of products,repacking, and even product maintenance. This means that warehouses needto be redesigned to adapt the requirements of such new services. The ARtechnology may be used to intuitively see the effect of re-planning froma global perspective, so that interactive digital models to be changedare placed in an existing real warehouse environment. A manager mayexamine whether a planned change has an appropriate size, andestablishes a model for a new work procedure. Benefiting from this, realwarehouses in the future may be used as test sites for the planning ofwarehouse operations.

During transportation optimization, the application of high and newinformation technologies by logistics achieves much more timely,reliable, and safe cargo transportation. In the fields such ascompleteness check, international trade, driver navigation, and cargoloading, the AR display technology provided in the embodiments of thepresent disclosure has the potential of further optimizing cargotransportation, thereby achieving more efficient sorting. A pickerwearing an AR device can know whether a load is complete with a quickscan. At present, the work requires manual statistics, or it takes along time for a worker to scan barcodes one by one with a handhelddevice. A wearable AR device can use a combination of a scanner and a 3Ddepth-of-field sensor to determine a quantity of pallets or packages (byscanning a special label on each package) or determine the volume of apackage (by using a measurement device). A measured value is comparedwith a predefined value, and a result is presented to a picker beforethe eyes. Such an AR system may also scan objects for damage or errors.

It may be understood that the foregoing six specific scenarios are notused to limit the embodiments of the present disclosure, but instead areused to help understand a provided executable environment. In operation,the method for AR display provided in the present disclosure may furtherbe applied to gaming, medical training, and the like. For example, bymeans of the method for AR display, users can better immerse themselvesin gaming environments, or interns can observe surgery closely withoutentering surgeries, so as to learn more practical knowledge. Inaddition, the method for AR display may also be used by real estateagents to show clients in air-conditioned offices pictures presented byusing the method for AR display, instead of taking clients around to seehouses in bad weather.

An AR-based interface is not limited to a computer device. The interfacecan also be used to control cars, entertainment devices, and householdfittings such as a heating system.

The method for determining information of an image capturing device inthe embodiments of the present disclosure is described below. Referringto FIG. 4, the method for determining information of an image capturingdevice in the embodiments of the present disclosure includes thefollowings.

101. An image capturing device acquires a first image of a targetobject.

In one embodiment, the image capturing device acquires a target objectfrom different photographing angles in the same real scene. The targetobject may be a book, a tree, a television set or a can of cola. This isnot limited herein. It is intended to integrate a virtual object withthe target object. The virtual object may be a cartoon characterdesigned beforehand by a designer or a doodle. This is not limitedherein.

102. The image capturing device extracts coordinate information of afirst 2D key point corresponding to the target object from the firstimage.

In one embodiment, the image capturing device first acquires a first 2Dkey point in the first image and corresponding coordinate information. Amanner based on area division may be used to detect key points.Generally, one frame of image may be divided into 80×60 regular areas,and correspondence between key points and areas are established.

A features from accelerated segment test (FAST) manner or a HarrisCorner Detection manner may be used to acquire a plurality of first 2Dkey points from the first image.

In operation, key points may be replaced with lines or edges. Points areused as an example for description in this solution. However, this doesnot constitute a limitation to the embodiments of the presentdisclosure.

103. The image capturing device acquires a second image of the targetobject.

In one embodiment, the image capturing device further acquires thesecond image. It may be understood that the second image and the firstimage come from the same target object. To be specific, the first imageand the second image are two different pictures obtained byphotographing a target object.

104. The image capturing device acquires coordinate information of asecond 2D key point in the second image according to the coordinateinformation of the first 2D key point, the first 2D key point and thesecond 2D key point being the same key point in different images, andthe first image being any frame of image before the second image.

In one embodiment, when a new frame of image is loaded, the image may beconsidered as the second image. In other words, the second image is anew frame of image that appears after the first image. After first 2Dkey points are obtained from the first image, a key point trackingalgorithm (for example, a Lucas-Kanade pyramidal tracking algorithm) isused to calculate 2D positions of these first 2D key points in thecurrent second image, so that the coordinate information of the second2D key points is obtained.

It may be understood that the first 2D key point and the second 2D keypoint are physically the same key point. However, because the positionchanges, corresponding coordinate information also changes, and “first”and “second” are thus used to express the same key point.

Similarly, when a new image is loaded subsequently, the key pointtracking algorithm still needs to be used to calculate a 2D position ofa 2D key point in a previous frame of image in the new loaded image. Inthis way, the processing manner is similar for the k^(th) image. Detailsare therefore not described herein.

105. The image capturing device acquires coordinate information of a 3Dkey point corresponding to the target object, the 3D key point being akey point obtained after triangulation processing.

In one embodiment, the image capturing device needs to acquire a 3D keypoint, and may determine coordinate information of the 3D key point. Thecoordinate information of the 3D key point mainly includes coordinateinformation in six dimensions, that is, displacements in threedirections and three angles.

The 3D key point is a key point on which triangulation processing hasbeen successfully performed in previous frames of image.

106. The image capturing device determines camera posture information ofthe second image according to the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key point, andthe coordinate information of the 3D key point, the camera postureinformation being used for determining a position of the target objectin the second image.

In one embodiment, the image capturing device may acquire 2D-2Dcorrespondences according to the coordinate information of the first 2Dkey point and the coordinate information of the second 2D key point,acquire 3D-2D correspondences according to the coordinate information ofthe 3D key point, and determine the camera posture information of thesecond image by using both the 2D-2D correspondences and the 3D-2Dcorrespondences. The camera posture information is used for determininga position of the target object in the second image. The camera postureinformation may represent photographing space information of the secondimage. The target object is usually a photographed physical object. Inaddition, the camera posture information is also used to acquire acorresponding 3D key point and coordinate information of the 3D keypoint from an image loaded subsequently.

Thus, the embodiments of the present disclosure provide a method fordetermining information of an image capturing device. In the method, theimage capturing device first acquires coordinate information of a first2D key point in a first image, the image capturing device then acquirescoordinate information of a second 2D key point in a second imageaccording to the coordinate information of the first 2D key point, thefirst 2D key point and the second 2D key point being the same key pointin different images, and the first image being any frame of image beforethe second image, the image capturing device then acquires coordinateinformation of a 3D key point, and the image capturing device finallydetermines camera posture information of the second image according tothe coordinate information of the first 2D key point, the coordinateinformation of the second 2D key point, and the coordinate informationof the 3D key point. The camera posture information is used fordetermining a position of the target object in the second image. Bymeans of the foregoing manners, the impact of the coordinate informationof the 3D key point and the coordinate information of the second 2D keypoint on the camera posture information are considered, and thecoordinate information of the first 2D key point is also considered. Inother words, 3D-2D correspondences and 2D-2D correspondences arecombined in this solution, to obtain more key points during theestimation of camera posture information of the second image, so that inone aspect, errors in camera posture information are reduced, and inanother aspect, a tracking loss because of an excessively small quantityof key points in tracking can be avoided.

Optionally, based on the embodiment corresponding to FIG. 4, in oneembodiment, before the acquiring, by the image capturing device,coordinate information of a second 2D key point in the second imageaccording to the coordinate information of the first 2D key point, themethod may further include the following steps: determining, by theimage capturing device, whether a quantity of second 2D key points inthe second image reaches a threshold; and when the quantity of thesecond 2D key points reaches the threshold, triggering the operation ofacquiring, by the image capturing device, coordinate information of asecond 2D key point in the second image according to the coordinateinformation of the first 2D key point.

In one embodiment, a 2D-2D key point tracking algorithm used herein is arelatively mature and stable algorithm. If it is found that insufficientkey points are successfully tracked, the system determines that trackingfails.

Specifically, the image capturing device first presets a threshold, forexample, 50, and then determines whether a quantity of the key pointsfrom the first image to the second image reaches the threshold. Inoperation, there may be two determination manners. In the first manner,it is assumed that a number of the first 2D key points in the firstimage is 100, and the number of the second 2D key points in the secondimage is 70. The difference between the two numbers is 30. It isdetermined that the difference is less than the preset threshold. In thesecond manner, the number of the second key points in the second imageis directly acquired. It is simpler to implement the second manner thanthe first manner because it is not necessary to count the key points inthe two images, so that the system efficiency is improved.

In a case that the number of the second 2D key points reaches thethreshold, the coordinate information of these second 2D key points mayfurther be acquired.

It may be understood that when a new image is loaded subsequently, it isstill necessary to detect whether the number of the key points that aresuccessfully tracked is large enough (that is, compared with the presetthreshold). If there are sufficient key points, the coordinateinformation of the key points continues to be acquired.

Next, in one embodiment of the present disclosure, before acquiring thecoordinate information of the second 2D key points, the image capturingdevice needs to acquire the second 2D key points first, determineswhether the number of the second 2D key points reaches the threshold,and continues to acquire the coordinate information of the second 2D keypoints when the number reaches the threshold. In the foregoing manner,it can be ensured that the number of the acquired second 2D key pointsreaches the threshold, to avoid a tracking loss because of anexcessively small number of second 2D key points in tracking, therebyimproving the practicability and feasibility of the solution.

Optionally, based on FIG. 4, in another embodiment, the acquiring, bythe image capturing device, coordinate information of a second 2D keypoint in the second image according to the coordinate information of thefirst 2D key point may further include the following steps: determiningthe first 2D key point from the first image; determining the second 2Dkey point corresponding to the first 2D key point from the first image;and determining the coordinate information of the second 2D key pointaccording to the coordinate information of the first 2D key point anddisplacement information, where the displacement information is adistance from the first 2D key point to the second 2D key point.

In one embodiment, how to acquire the coordinate information of thesecond 2D key point in the second image is described. Specifically, inone manner of acquiring the coordinate information of the second 2D keypoint, the first 2D key point is first obtained from the first image.Certainly, there are usually a plurality of first 2D key points. Forease of understanding, one of the first 2D key points may be used as anexample for description.

In one embodiment, a Lucas-Kanade pyramidal tracking algorithm may beused to determine a coordinate position of the second 2D key point. Itis assumed that there is a pyramid in the first image. The tip of thepyramid is the first 2D key point. The coordinate information of thefirst 2D key point is (a, b). Next, the camera displaces after onesecond, and the second image is obtained. The tip of the pyramid isstill present in the second image, but has also displaced. Thedisplacement includes an upward movement by α and a leftward movement byβ. In this case, it can be obtained by using the displacementinformation that the coordinate information of the second 2D key pointis (a+α, b−β).

It may be understood that the coordinate information is only an example.In operation, the coordinate information may be represented in anothermanner, for example, by an angle. In addition, the coordinateinformation also has a corresponding representation manner. This is notlimited herein.

Next, in one embodiment of the present disclosure, in the process ofacquiring the coordinate information of the second 2D key point in thesecond image, the first 2D key point first needs to be determined fromthe first image, the second 2D key point is found from the second image,and the displacement information between the two key points is used tocalculate the coordinate information of the second 2D key point. In theforegoing manner, the second 2D key point can be accurately located, toprovide a feasible method for implementing the solution, therebyimproving the practicability and operability of the solution.

Optionally, based on FIG. 4, in another embodiment, after determining,by the image capturing device, whether a number of second 2D key pointsin the second image reaches a threshold, the method may further includethe following steps: when the number of the second 2D key points doesnot reach the threshold, acquiring, by the image capturing device, n keypoints from the first image, to enable a sum of the number of the second2D key points and a number of the n key points to reach the threshold,where n is a positive integer, the n key points are acquired from atarget area corresponding to the first image, the first image includes aplurality of equally divided areas, and a number of the key points inthe target area is less than a preset key point threshold; andacquiring, by the image capturing device, coordinate information of then key points and coordinate information of the second 2D key points inthe second image according to the coordinate information of the first 2Dkey point and coordinate information of the n key points.

It may be understood that when a new image is loaded subsequently, it isstill necessary to detect whether the number of the key points that aresuccessfully tracked is large enough (that is, compared with the presetthreshold). If there are insufficient key points, it is necessary todetect some more key points in areas with a relatively small number ofcorresponding key points (key points are detected in a previous frame ofimage of the current frame of image), and the current frame of image istracked from the key points in the previous frame. In this way, thek^(th) frame image has a similar processing manner.

Next, in one embodiment of the present disclosure, in the foregoingmanner, it can be ensured that the number of the acquired second 2D keypoints reaches the threshold, to avoid a tracking loss because of anexcessively small number of second 2D key points in tracking, therebyimproving the flexibility and feasibility of the solution.

Optionally, based on FIG. 4, in another embodiment, the acquiring, bythe image capturing device, coordinate information of the second 2D keypoints in the second image according to the coordinate information ofthe first 2D key point and coordinate information of the n key pointsincludes the following steps: determining the first 2D key point and then key points from the first image; determining the second 2D key pointscorresponding to the first 2D key point and the n key points from thefirst image; and determining the coordinate information of the n keypoints and the coordinate information of the second 2D key points in thesecond image according to the coordinate information of the first 2D keypoint, the coordinate information of the n key points, and displacementinformation, where the displacement information includes a distance fromthe first 2D key point to the second 2D key point and distances from then key points to the second 2D key points.

In one embodiment, how to acquire the coordinate information of thesecond 2D key point in the second image and the coordinate informationof n key points is described. Specifically, in one manner of acquiringthe coordinate information of the second 2D key point, the first 2D keypoint is first obtained from the first image. Certainly, there areusually a plurality of first 2D key points. Meanwhile, n key points areacquired from the first image, where n is a positive integer.

It is assumed that one 2D key point is found in the first image. The 2Dkey point is any one of the first 2D key point and the n key points.Similarly, by means of the second optional embodiment corresponding toFIG. 4, the coordinate information of the second 2D key point may bedetermined according to the displacement information and the key pointinformation. This is not described in detail herein.

Optionally, based on FIG. 4, in another, after the determining thecamera posture information of the second image, the method may furtherinclude the following steps: performing, by the image capturing device,triangulation processing on the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key points,camera posture information of the first image, and the camera postureinformation of the second image, and obtaining a 3D key point to beselected and coordinate information of the 3D key point to be selected;determining, by the image capturing device, an observation angleaccording to the 3D key point to be selected, where the observationangle is an angle between a connecting line between a center point inthe first image and the 3D key point to be selected and a connectingline between a center point in the second image and the 3D key point tobe selected; and when the observation angle is greater than a presetangle, determining, by the image capturing device, the coordinateinformation of the 3D key point to be selected as coordinate informationof a target 3D key point.

In one embodiment, how to determine a target 3D key point in asubsequent image and coordinate information corresponding to the target3D key point is described.

Specifically, the coordinate information of the first 2D key point andthe coordinate information of the second 2D key point are firstacquired, the pre-obtained camera posture information corresponding tothe first image is then acquired, and the camera posture informationcorresponding to the second image obtained in step 104 is acquired. Thefour pieces of information are combined to perform triangulationprocessing, and the target 3D key point is obtained.

Herein, an observation angle in triangulation needs to be estimated. Forease of understanding, referring to FIG. 5, FIG. 5 is a schematicdiagram of determining an observation angle according to an embodimentof the present disclosure. As shown in FIG. 5, the observation angle isan angle of observing the target 3D key point from the positions of thecenter points of two images. In a case that most of the observationangles corresponding to these target 3D key points are excessivelysmall, it may indicate that current initialization has relatively largeerrors. In this case, the system needs to be labeled as “initializationfails”. Initialization is attempted again when a subsequent frame isloaded subsequently.

Next, in one embodiment of the present disclosure, in the manner inwhich the image capturing device determines the coordinate informationof a 3D key point, coordinate information of a 3D key point to beselected may be obtained first, and an observation angle of the 3D keypoint to be selected is then obtained. In a case that the observationangle is greater than a preset angle, the coordinate information of the3D key point to be selected is determined as the coordinate informationof the target 3D key point. In the foregoing manner, a target 3D keypoint whose observation angle is greater than a preset angle may beselected. In a case that an observation angle is large, it may indicatethat the initialization of a SLAM system has a small error, therebyimproving the practicability and operability of the solution.

Optionally, based on FIG. 4, in another embodiment, after the acquiring,by the image capturing device, coordinate information of a second 2D keypoint in the second image according to the coordinate information of thefirst 2D key point, the method may further include the following steps:generating, by the image capturing device, a tracking record, where thetracking record is used for recording coordinate information of 2D keypoints in a plurality of images, the first image in the tracking recordis the first image, the last image in the tracking record is the secondimage, and at least one image is included between the first image andthe second image.

In one embodiment, how to generate a tracking record corresponding to akey point is described in detail. For ease of understanding, referringto FIG. 6, FIG. 6 is a schematic diagram of generating a tracking recordaccording to an embodiment of the present disclosure. As shown in thefigure, four frames of image are shown, and are a t^(th) frame of image,a (t+1)^(th) frame of image, a (t+2)^(th) frame of image, and a(t+3)^(th) frame of image. A solid star represents a key point oftracking, a hollow star represents a starting point of tracking, and adotted line represents a tracking record.

In the system, starting from a detected key point in a frame, n framesof image are continuously tracked until the tracking is lost, and anentire obtained trajectory is referred to as one tracking record. Datasuch as frame numbers of images that the key point passes through, 2Dcoordinates of the key point in corresponding frames, whethertriangulation succeeds, whether the key point has a corresponding 3Dpoint, and the largest observation angle of the key point observed indifferent frames is recorded in the data structure in each trackingrecord.

The tracking record is related to the first image and the second image.Herein, the first image is the first image in the tracking record, andthe second image is the last image in the tracking record. Therefore, itmay be considered that at least one image is included between the firstimage and the second image.

Next, in the foregoing manner, a tracking record including coordinateinformation of a 2D key point in a plurality of images can be obtained.The tracking record can track a 2D key point that passes through theimages, so that it is convenient to rapidly and accurately acquire fromthe tracking record the coordinate information of a 2D key pointcorresponding to required images, thereby improving the feasibility ofthe solution.

Optionally, based on FIG. 4, in another embodiment, the image capturingdevice acquires coordinate information of a 3D key point correspondingto the target object may include the following steps: acquiringcoordinate information of a third 2D key point of the target object andcoordinate information of a fourth 2D key point of the target objectaccording to the tracking record, where the third 2D key point belongsto a third image, the fourth 2D key point belongs to a fourth image, andthe third image and the fourth image are both located between the firstimage and the second image; performing triangulation processing on thecoordinate information of the first 2D key point and the coordinateinformation of the third 2D key point, and obtaining coordinateinformation of a first 3D key point; performing triangulation processingon the coordinate information of the third 2D key point and thecoordinate information of the fourth 2D key point, and obtainingcoordinate information of a second 3D key point; determining a firstobservation angle according to the coordinate information of the first3D key point, and determining a second observation angle according tothe coordinate information of the second 3D key point, where the firstobservation angle is an angle between a connecting line between a centerpoint in the first image and the first 3D key point and a connectingline between a center point in the third image and the first 3D keypoint, and the second observation angle is an angle between a connectingline between a center point in the second image and the second 3D keypoint and a connecting line between a center point in the third imageand the second 3D key point; when the first observation angle is greaterthan the second observation angle, determining the coordinateinformation of the first 3D key point as the coordinate information ofthe 3D key point of the target object; and when the second observationangle is greater than the first observation angle, determining thecoordinate information of the second 3D key point as the coordinateinformation of the 3D key point of the target object.

In one embodiment, how to use a tracking record to obtain an optimal 3Dkey point is described. Generally, a tracking record includesinformation of a plurality of frames of image. Therefore, a 3D key pointis calculated between every two images, and an optimal 3D key point isfinally selected. The plurality of frames of image herein may be keyframes of image. The SLAM system uses a technology based on key framesto reduce a data amount of historical data to be maintained. Key framesmay be selected and managed in a simple regular frame skipping manner.For example, one of every m frames is selected as a key frame.Certainly, some simple heuristic checks need to be performed on the keyframes. Only key frames with a relatively large amount of informationare kept.

Specifically, the coordinate information of the third 2D key point andthe coordinate information of the fourth 2D key point are acquiredaccording to the tracking record, where the third 2D key point belongsto a third image, the fourth 2D key point belongs to a fourth image, andthe third image and the fourth image are both located between the firstimage and the second image. Next, triangulation processing is performedon the coordinate information of the first 2D key point, the coordinateinformation of the third 2D key point, camera posture information of thefirst image, and camera posture information of the third image, and thecoordinate information of the first 3D key point is obtained.Triangulation processing is performed on the coordinate information ofthe third 2D key point, the coordinate information of the fourth 2D keypoint, camera posture information of the third image, and camera postureinformation of the fourth image, and the coordinate information of thesecond 3D key point is obtained. In operation, triangulation processingcan also be performed on the coordinate information of the first 2D keypoint and the coordinate information of the fourth 2D key point, and thecoordinate information of the third 3D key point is obtained. At thispoint, the coordinate information of a 3D key point corresponding toevery two images is obtained.

In this case, according to the method for determining an observationangle described in the fifth optional embodiment corresponding to FIG.4, a first observation angle corresponding to a first 3D key point, asecond observation angle corresponding to a second 3D key point, and athird observation angle corresponding to a third 3D key point may beseparately obtained. The first observation angle, the second observationangle, and the third observation angle are compared to select thelargest observation angle. A 3D key point corresponding to the largestobservation angle is the required 3D key point.

It may be understood that one embodiment is described by using anexample of comparing three observation angles. However, this does notconstitute any limitation on this solution.

Again, in one embodiment of the present disclosure, in the process inwhich the image capturing device determines the coordinate informationof the 3D key point, a tracking record is used to separately acquire 2Dkey points in a plurality of frames of image, triangulation processingis then performed on 2D key points in every two images to obtain aplurality of 3D key points. These 3D key points separately correspond toone observation angle. A 3D key point with the largest observation angleis finally selected. In the foregoing manner, the 3D key pointcorresponding to the largest observation angle may be obtained accordingto the tracking record. In a case that an observation angle is large, itmay indicate that the initialization of a SLAM system has a small error,thereby improving the practicability and operability of the solution.

Optionally, based on FIG. 4, in another embodiment, the determiningcamera posture information of the second image according to thecoordinate information of the first 2D key point, the coordinateinformation of the second 2D key point, and the coordinate informationof the 3D key point may include the following steps: acquiring, by theimage capturing device, a 3D-2D projection error according to thecoordinate information of the 3D key point; acquiring, by the imagecapturing device, a 2D-2D epipolar distance according to the coordinateinformation of the first 2D key point and the coordinate information ofthe second 2D key point; and calculating, by the image capturing device,a target function according to the 3D-2D projection error and the 2D-2Depipolar distance, and finding the minimum of the target function, toobtain the camera posture information of the second image.

In one embodiment, how to acquire the camera posture information of thesecond image is described.

It may be understood that herein, the first image may be the first frameof image in the tracking record, and the second image may be the lastframe of image in the tracking record.

A projection error is often used in computer vision. For example, duringthe calculation of a planar homography matrix and a projection matrix, areprojection error is usually used to build a cost function, and thecost function is then minimized to optimize the homography matrix orprojection matrix. The projection error is used because it takes both acalculation error of a homography matrix and a measurement error of animage point into consideration. Therefore, the precision of using theprojection error is higher than the precision of using a one-sided erroror a symmetric transfer error.

Originating from a structure from motion problem, an epipolar distanceis a special geometric relationship that exists between two imagesgenerated at two camera positions, and is a basic model for solving a2D-2D camera pose between frames. With an epipolar distance model, the2D-2D camera pose may be solved by using the following process.

A plurality of corresponding points are used to perform frame matching,to calculate an elementary matrix. The elementary matrix is thendecomposed to calculate an extrinsic parameter, to obtain a rotationamount and a translation amount, which may also be considered as acamera pose.

Furthermore, in one embodiment of the present disclosure, more accuratecamera posture information can be obtained in the foregoing manner. Thecamera posture information is a minimum solution value of the targetfunction and may be considered as an optimal solution. Therefore, theposition of the target object in the second image can be more accurate.

Optionally, based on FIG. 4, in another embodiment, the acquiring, bythe image capturing device, a 3D-2D projection error according to thecoordinate information of the 3D key point in the second image and thecoordinate information of the second 2D key point, the method mayfurther include the following steps:

calculating the 3D-2D projection error by using the following formula:

${A = {\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}}},$

where A denotes the 3D-2D projection error, C_(3D2D) denotes a set of 3Dkey points projected on the second image, u_(i) denotes the coordinateinformation of the second 2D key point, π_(x) denotes a projectionfunction for 3D-2D projection, the projection function and the cameraposture information have an association relationship, and p_(i) denotesthe coordinate information of the 3D key point; the acquiring a 2D-2Depipolar distance according to the coordinate information of the first2D key point and the coordinate information of the second 2D key pointincludes:

calculating the 2D-2D epipolar distance by using the following formula:

${B = {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}},$

where B denotes the 2D-2D epipolar distance, C_(2D2D) denotes a set ofthe second 2D key points and the first 2D key points, λ denotes a presetequilibrium constant, the preset equilibrium constant is used forkeeping a difference between 2D-2D epipolar distance and the 3D-2Dprojection error within a preset range, u′_(i) denotes the coordinateinformation of the first 2D key point, F_(x) denotes an elementarymatrix between the second image and the first image, and the projectionfunction and the camera posture information have an associationrelationship; and the calculating a target function according to the3D-2D projection error and the 2D-2D epipolar distance may include:

calculating the target function by using the following formula:

${{E(x)} = {{\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}} + {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}}},$

where E(x) denotes the target function, and x denotes the camera postureinformation of the second image.

In one embodiment, 3D is the previous 3D key points that have beensuccessfully triangulated, and 2D is 2D image coordinates in the currentframe that correspond to those 3D key points observed in the currentframe. A new target function is proposed herein and includes both a3D-2D projection error and a 2D-2D epipolar distance. In this way, thenew target function is minimized to solve the camera postureinformation.

It is assumed that the camera posture information to be solved is x,where x includes a rotation amount R and a translation amount t. Thetarget function is as follows (Note that in the following thecoordinates of both a 2D key point and a 3D key point are in the form ofhomogeneous coordinates. For example, the homogeneous coordinates of a2D key point [x,y]^(T) are [x,y,1]^(T) the homogeneous coordinates of a3D key point [x,y,z]^(T) are [x,y,z,1]^(T). The symbol T hereinrepresents the transpose of a matrix or vector.

First, the 3D-2D projection error is calculated by using the followingformula:

$\begin{matrix}{{A = {\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}}},} & (1)\end{matrix}$

where, A denotes the 3D-2D projection error, ∥ ∥₂ denotes the norm of 2,C_(3D2D) denotes a set of 3D key points projected on the second image,denotes the coordinate information of the second 2D key point, π_(x)denotes a projection function for 3D-2D projection, the projectionfunction and the camera posture information have an associationrelationship, and p_(i) denotes the coordinate information of the 3D keypoint.

Next, the 2D-2D epipolar distance is calculated by using the followingformula:

$\begin{matrix}{{B = {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}},} & (2)\end{matrix}$

where B denotes the 2D-2D epipolar distance, C_(2D2D) denotes a set ofthe second 2D key points and the first 2D key points, λ denotes a presetequilibrium constant, the preset equilibrium constant is used forkeeping a difference between 2D-2D epipolar distance and the 3D-2Dprojection error within a preset range, u′_(i) denotes the coordinateinformation of the first 2D key point, F_(x) denotes an elementarymatrix between the second image and the first image, and the projectionfunction and the camera posture information have an associationrelationship.

The algorithm (v)_(j) in Formula (2) represents taking the j^(th)element in the vector v.

Finally, Formula (1) and Formula (2) are combined to calculate thetarget function:

$\begin{matrix}{{E(x)} = {{\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}} + {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}}} & (3)\end{matrix}$

where E (x) denotes the target function, and x denotes the cameraposture information of the second image.

Formula (2) is in fact an approximate epipolar distance named a Sampsondistance. λ in Formula (3) is a weight value for balancing Formula (1)and Formula (2), and may be selected according to an actual experimentaleffect. The target function is a form based on a quadratic cost. Toachieve a more stable effect, the target function may be robustified inactual use. For example, a Huber function is used to separatelyrobustify two items of the target function.

The minimum of the target function is found to make the solution valid.x is parameterized, and a non-linear least square method(Levenberg-Marquardt, LM) is used to find the solution. During theoptimized solution finding, a rough initial value of x is first obtainedby using another method, and iterative optimization is then performed toavoid falling into a local optimal solution.

Furthermore, in one embodiment of the present disclosure, specificcalculation formulas are provided. The 3D-2D projection error and the2D-2D epipolar distance are separately calculated according tocorresponding formulas, and are combined to obtain the target function.In the foregoing manner, the reliability of obtaining a target functioncan be improved, thereby improving the feasibility and practicability ofthe solution.

For ease of understanding, the following may describe a process ofdetermining camera posture information in the embodiments of the presentdisclosure in detail by using a specific application scenario. Referringto FIG. 7, FIG. 7 is a schematic flowchart of determining camera postureinformation in an application scenario according to an embodiment of thepresent disclosure. The procedure may include the followings.

A201. Start loading a new frame of image first.

A202. Determine whether the loaded new frame of image is the first frameof image, and if yes, perform B202, or if not, perform B201.

B201. Continue to determine whether a key point in a previous frame ofimage reaches a threshold, and if yes, perform B204, or if not, performB203.

B202. In the SLAM system in one embodiment of the present disclosure,manage the detection and tracking of key points based on areas andtracking records. For example, starting from the first frame of imageI1, significant key points are detected from the image first, the imageis then divided into regular areas, and a correspondence between the keypoints and areas in which the key points are located is established.

B203. After a new frame of image I3 is loaded, it is necessary to firstdetect whether the number of the key points that are successfullytracked in the previous frame I2 is large enough (that is, compared withthe preset threshold). If there are insufficient key points, it isnecessary to detect some more key points in areas with a relativelysmall number of corresponding key points. Key points are detected in I2herein.

B204. Track the current frame I3 from key points in I2. In this way, thek^(th) frame image has a similar processing manner. After a new frame ofimage I2 is loaded, for key points in the image I1, a 2D-2D key pointtracking algorithm is used to calculate 2D positions of these key pointsin the current frame I2.

B205. Determine whether a number of the key points in a current imagereaches the threshold, and if yes, perform C201, or if not, performB206.

B206. Determine that the system fails tracking.

C201. Further determine whether the system is initialized, and if yes,perform D201, or if not, perform C202.

C202. A classic homography-based method is used as the initializationmethod herein. Generally, before the system has been initialized, thecoordinates of 2D key points in the current frame that are ofsuccessfully tracked key points in the current frame and the coordinatesof 2D key points in the first frame that are of the successfully trackedkey points as well as a Random Sample Consensus (RANSAC) algorithm areused to calculate a homography matrix. The homography matrix is thendecomposed to obtain a correlated photographing pose (correlatedlocation information) of the current frame relative to the first frame.Finally, the obtained correlated photographing pose is used to performtriangulation on 2D-2D correspondences to obtain the coordinates of thecorresponding 3D key points. Herein, it is necessary to estimateobservation angles in triangulation. In a case that most of theobservation angles corresponding to these 3D key points are excessivelysmall, it may indicate that current initialization has relatively largeerrors. In this case, the system needs to be labeled as “initializationfails”. Initialization is attempted again when a subsequent frame isloaded subsequently.

C203. Determine whether initialization succeeds, and if yes, performC204, or if not, perform A201.

C204. Label that the system has been successfully initialized.

D201. Propose a new target function that includes both a 3D-2Dprojection error and a 2D-2D epipolar distance herein. In this way, thenew target function is minimized to solve the camera postureinformation.

D202. Output online the camera posture information obtained in D201.

E201. The system uses a technology based on key frames to reduce a dataamount of historical data to be maintained (Key frames may be selectedand managed in a simple regular frame skipping manner. For example, oneof every m frames is selected as a key frame. Certainly, some simpleheuristic checks need to be performed on the key frames. Only key frameswith a relatively large amount of information are kept.). In a keyframe, a tracking record that has not been triangulated is triangulatedby using a starting point and an end point of the tracking record, so asto calculate 3D key points corresponding to the tracking record.Observation angles of the 3D key points are checked, and 3D points witha relatively large observation angle are kept. This step may beperformed for every frame instead of being performed for only keyframes. This may be adjusted according to an actual calculation speed.

F201. Add local or global bundle adjustment and mechanisms such asloop-closure detection and correction to the system to improve theprecision of the entire system. These modules are relatively independentand may be added flexibly similar to an existing SLAM system.

The image capturing device in the embodiments of the present disclosureare described below in detail. Referring to FIG. 8, the image capturingdevice 30 in one embodiment of the present disclosure includes one ormore processors and one or more memories storing program units/modules,the program units being executed by the processor, and the program unitsinclude: an acquisition module 301, an extraction module 302, and adetermining module 303.

The acquisition module 301 is configured to obtain a first image of atarget object. The extraction module 302 is configured to extractcoordinate information of a first 2D key point corresponding to thetarget object from the first image acquired by the acquisition module301.

Further, the acquisition module 301 is configured to acquire a secondimage of the target object. The acquisition module 301 may further beconfigured to acquire coordinate information of a second 2D key point inthe second image according to the coordinate information of the first 2Dkey point extracted by the extraction module 302, the first 2D key pointand the second 2D key point being the same key point in the targetobject. The acquisition module 301 may also further be configured toacquire coordinate information of a 3D key point corresponding to thetarget object, the 3D key point being a key point obtained aftertriangulation processing.

The determining module 303 is configured to determine camera postureinformation of the second image according to the coordinate informationof the first 2D key point extracted by the extraction module 302 and thecoordinate information of the second 2D key point and the coordinateinformation of the 3D key point that are acquired by the acquisitionmodule 301, the camera posture information being used for determining aposition of the target object in the second image.

Optionally, based on FIG. 8, referring to FIG. 9, in another embodiment,the program units further includes: a judgment module 304, and a triggermodule 305.

The judgment module 304 is configured to: before the acquisition module301 acquires the coordinate information of the second 2D key point inthe second image according to the coordinate information of the first 2Dkey point, determine whether a number of second 2D key points in thesecond image reaches a threshold; and

The trigger module 305 is configured to: when the judgment module 304determines that the number of the second 2D key points reaches thethreshold, trigger the operation of acquiring the coordinate informationof the second 2D key point in the second image according to thecoordinate information of the first 2D key point.

Optionally, based on FIG. 8 or FIG. 9, in another embodiment, thedetermining module 303 is specifically configured to: determine thefirst 2D key point from the first image; determine the second 2D keypoint corresponding to the first 2D key point from the first image; anddetermine the coordinate information of the second 2D key pointaccording to the coordinate information of the first 2D key point anddisplacement information, where the displacement information is adistance from the first 2D key point to the second 2D key point.

Optionally, based on FIG. 9, in another embodiment, when the number ofthe second 2D key points does not reach the threshold, the acquisitionmodule 301 is further configured to: acquire n key points from the firstimage, to enable a sum of the number of the second 2D key points and anumber of the n key points to reach the threshold, where n is a positiveinteger, the n key points are acquired from a target area correspondingto the first image, the first image includes a plurality of equallydivided areas, and a number of the key points in the target area is lessthan a preset key point threshold; and acquire coordinate information ofthe n key points and coordinate information of the second 2D key pointsin the second image according to the coordinate information of the first2D key point and coordinate information of the n key points.

Optionally, based on FIG. 9, in another embodiment, the determiningmodule 303 is specifically configured to: determine the first 2D keypoint and the n key points from the first image; determine the second 2Dkey points corresponding to the first 2D key point and the n key pointsfrom the first image; and determine the coordinate information of the nkey points and the coordinate information of the second 2D key points inthe second image according to the coordinate information of the first 2Dkey point, the coordinate information of the n key points, anddisplacement information, where the displacement information includes adistance from the first 2D key point to the second 2D key point anddistances from the n key points to the second 2D key points.

Optionally, based on FIG. 8, in another embodiment, the determiningmodule 302 is further configured to: after the camera postureinformation of the second image is determined, perform triangulationprocessing on the coordinate information of the first 2D key point, thecoordinate information of the second 2D key points, camera postureinformation of the first image, and the camera posture information ofthe second image, and obtain a 3D key point to be selected andcoordinate information of the 3D key point to be selected; determine anobservation angle according to the 3D key point to be selected, wherethe observation angle is an angle between a connecting line between acenter point in the first image and the 3D key point to be selected anda connecting line between a center point in the second image and the 3Dkey point to be selected; and when the observation angle is greater thana preset angle, determine the coordinate information of the 3D key pointto be selected as coordinate information of a target 3D key point.

Next, in one embodiment of the present disclosure, in the manner inwhich the image capturing device determines the coordinate informationof a 3D key point, triangulation processing may be first performed onthe coordinate information of the first 2D key point and the coordinateinformation of the second 2D key point, coordinate information of a 3Dkey point to be selected may be obtained, and an observation angle ofthe 3D key point to be selected is then obtained. In a case that theobservation angle is greater than a preset angle, the coordinateinformation of the 3D key point to be selected is determined as thecoordinate information of the target 3D key point. In the foregoingmanner, a target 3D key point whose observation angle is greater than apreset angle may be selected. In a case that an observation angle islarge, it may indicate that the initialization of a SLAM system has asmall error, thereby improving the practicability and operability of thesolution.

Optionally, based on FIG. 8, referring to FIG. 10, in anotherembodiment, the program units further includes a generation module 306.

The generation module 306 is configured to: after the acquisition module301 acquires the coordinate information of the second 2D key point inthe second image according to the coordinate information of the first 2Dkey point, generate a tracking record, where the tracking record is usedfor recording coordinate information of 2D key points in a plurality ofimages, the first image in the tracking record is the first image, thelast image in the tracking record is the second image, and at least oneimage is included between the first image and the second image.

Optionally, based on FIG. 10, in another embodiment, the acquisitionmodule 301 is specifically configured to: acquire coordinate informationof a third 2D key point of the target object and coordinate informationof a fourth 2D key point of the target object according to the trackingrecord, where the third 2D key point belongs to a third image, thefourth 2D key point belongs to a fourth image, and the third image andthe fourth image are both located between the first image and the secondimage; perform triangulation processing on the coordinate information ofthe first 2D key point and the coordinate information of the third 2Dkey point, and obtain coordinate information of a first 3D key point;perform triangulation processing on the coordinate information of thethird 2D key point and the coordinate information of the fourth 2D keypoint, and obtain coordinate information of a second 3D key point;determine a first observation angle according to the coordinateinformation of the first 3D key point, and determine a secondobservation angle according to the coordinate information of the second3D key point, where the first observation angle is an angle between aconnecting line between a center point in the first image and the first3D key point and a connecting line between a center point in the thirdimage and the first 3D key point, and the second observation angle is anangle between a connecting line between a center point in the secondimage and the second 3D key point and a connecting line between a centerpoint in the third image and the second 3D key point; when the firstobservation angle is greater than the second observation angle,determine the coordinate information of the first 3D key point as thecoordinate information of the 3D key point of the target object; andwhen the second observation angle is greater than the first observationangle, determine the coordinate information of the second 3D key pointas the coordinate information of the 3D key point of the target object.

Optionally, based on FIG. 8, FIG. 9, or FIG. 10, in another embodiment,the determining module 303 is specifically configured to: acquire a3D-2D projection error according to the coordinate information of the 3Dkey point; acquire a 2D-2D epipolar distance according to the coordinateinformation of the first 2D key point and the coordinate information ofthe second 2D key point; and calculate a target function according tothe 3D-2D projection error and the 2D-2D epipolar distance, and find theminimum of the target function, to obtain the camera posture informationof the second image.

Optionally, based on FIG. 8, FIG. 9, or FIG. 10, in another embodiment,the determining module 303 is specifically configured to calculate the3D-2D projection error by using the following formula:

${A = {\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}}},$

where A denotes the 3D-2D projection error, C_(3D2D) denotes a set of 3Dkey points projected on the second image, u_(i) denotes the coordinateinformation of the second 2D key point, π_(x) denotes a projectionfunction for 3D-2D projection, the projection function and the cameraposture information have an association relationship, and p_(i) denotesthe coordinate information of the 3D key point.

The determining module 303 is specifically configured to calculate the2D-2D epipolar distance by using the following formula:

${B = {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}},$

where B denotes the 2D-2D epipolar distance, C_(2D2D) denotes a set ofthe second 2D key points and the first 2D key points, λ denotes a presetequilibrium constant, the preset equilibrium constant is used forkeeping a difference between 2D-2D epipolar distance and the 3D-2Dprojection error within a preset range, u′_(i) denotes the coordinateinformation of the first 2D key point, F_(x) denotes an elementarymatrix between the second image and the first image, and the projectionfunction and the camera posture information have an associationrelationship.

The determining module 303 is specifically configured to calculate thetarget function by using the following formula:

${{E(x)} = {{\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}} + {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}}},$

where E(x) denotes the target function, and x denotes the camera postureinformation of the second image.

An embodiment of the present disclosure further provides anotherelectronic device. The electronic device may be, for example, an imagecapturing device. As shown in FIG. 11, for ease of description, onlyparts related to the embodiments of the present disclosure are shown.For specific technical details that are not disclosed, refer to themethod part in the embodiments of the present disclosure. The imagecapturing device may be a terminal. The terminal may be any terminaldevice including a mobile phone, a tablet computer, a personal digitalassistant (PDA), a point of sales (POS), and an on-board computer, andthe terminal being a mobile phone is used as an example.

FIG. 11 is a partial structural block diagram of a mobile phone relatedto a terminal according to an embodiment of the present disclosure.Referring to FIG. 11, the mobile phone includes components such as aradio frequency (RF) circuit 410, a memory 420, an input unit 430, adisplay unit 440, a sensor 450, an audio circuit 460, a Wi-Fi module470, a processor 480, and a power supply 490. A person skilled in theart may understand that the structure of the mobile phone shown in FIG.11 does not constitute a limitation on the mobile phone, and the mobilephone may include more components or fewer components than those shownin the figure, or some components may be combined, or a differentcomponent deployment may be used.

The following makes a specific description of components of the mobilephone with reference to FIG. 11.

The RF circuit 410 may be configured to receive and send signals duringan information receiving and sending process or a call process.Specifically, the RF circuit 410 receives downlink information from abase station, then delivers the downlink information to the processor480 for processing, and sends designed uplink data to the base station.Usually, the RF circuit 410 includes, but is not limited to, an antenna,at least one amplifier, a transceiver, a coupler, a low noise amplifier(LNA), and a duplexer. In addition, the RF circuit 410 may alsocommunicate with a network and another device by wireless communication.The wireless communication may use any communication standard orprotocol, including but not limited to Global System for Mobilecommunications (GSM), general packet radio service (GPRS), Code DivisionMultiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA),Long Term Evolution (LTE), email, Short Messaging Service (SMS), and thelike.

The memory 420 may be configured to store a software program and module.The processor 480 runs the software program and module stored in thememory 420, to implement various functional applications and dataprocessing of the mobile phone. The memory 420 may mainly include aprogram storage area and a data storage area. The program storage areamay store an operating system, an application program required by atleast one function (such as a sound playback function and an imagedisplay function), and the like. The data storage area may store data(such as audio data and an address book) created according to the use ofthe mobile phone, and the like. In addition, the memory 420 may includea high speed random access memory, and may also include a non-volatilememory, such as at least one magnetic disk storage device, a flashmemory, or another volatile solid-state storage device.

The input unit 430 may be configured to receive input digit or characterinformation, and generate a key signal input related to the user settingand function control of the mobile phone. Specifically, the input unit430 may include a touch panel 431 and another input device 432. Thetouch panel 431 may also be referred to as a touchscreen, and maycollect a touch operation of a user on or near the touch panel (such asan operation of a user on or near the touch panel 431 by using anysuitable object or attachment, such as a finger or a stylus), and drivea corresponding connection apparatus according to a preset program.Optionally, the touch panel 431 may include two parts, namely, a touchdetection apparatus and a touch controller. The touch detectionapparatus detects a touch position of the user, detects a signalgenerated by the touch operation, and transfers the signal to the touchcontroller. The touch controller receives the touch information from thetouch detection apparatus, converts the touch information into touchpoint coordinates, and sends the touch point coordinates to theprocessor 480. Moreover, the touch controller can receive and execute acommand sent from the processor 480. In addition, the touch panel 431may be implemented by using various types, such as a resistive type, acapacitance type, an infrared type, and a surface sound wave type. Inaddition to the touch panel 431, the input unit 430 may further includethe another input device 432. Specifically, the another input device 432may include, but is not limited to, one or more of a physical keyboard,a functional key (such as a volume control key or a switch key), a trackball, a mouse, and a joystick.

The display unit 440 may be configured to display information input bythe user or information provided for the user, and various menus of themobile phone. The display unit 440 may include a display panel 441.Optionally, the display panel 441 may be configured by using a liquidcrystal display (LCD), an organic light-emitting diode (OLED), or thelike. Further, the touch panel 431 may cover the display panel 441.After detecting a touch operation on or near the touch panel 431, thetouch panel 431 transfers the touch operation to the processor 480, todetermine a type of a touch event. Then, the processor 480 provides acorresponding visual output on the display panel 441 according to thetype of the touch event. Although in FIG. 11, the touch panel 431 andthe display panel 441 are used as two separate parts to implement inputand output functions of the mobile phone, in some embodiments, the touchpanel 431 and the display panel 441 may be integrated to implement theinput and output functions of the mobile phone.

The mobile phone may further include at least one sensor 450 such as anoptical sensor, a motion sensor, and other sensors. Specifically, theoptical sensor may include an ambient light sensor and a proximitysensor. The ambient light sensor may adjust luminance of the displaypanel 441 according to brightness of the ambient light. The proximitysensor may switch off the display panel 441 and/or backlight when themobile phone is moved to the ear. As one type of motion sensor, anacceleration sensor can detect magnitude of accelerations in variousdirections (generally on three axes), may detect magnitude and adirection of the gravity when static, and may be applied to anapplication that recognizes the attitude of the mobile phone (forexample, switching between landscape orientation and portraitorientation, a related game, and magnetometer attitude calibration), afunction related to vibration recognition (such as a pedometer and aknock), and the like. Other sensors, such as a gyroscope, a barometer, ahygrometer, a thermometer, and an infrared sensor, which may beconfigured in the mobile phone, are not further described herein.

The audio circuit 460, a speaker 461, and a microphone 462 may provideaudio interfaces between the user and the mobile phone. The audiocircuit 460 may convert received audio data into an electrical signaland transmit the electrical signal to the speaker 461. The speaker 461converts the electrical signal into a sound signal for output. On theother hand, the microphone 462 converts a collected sound signal into anelectrical signal. The audio circuit 460 receives the electrical signal,converts the electrical signal into audio data, and outputs the audiodata to the processor 480 for processing. Then, the processor 480 sendsthe audio data to, for example, another mobile phone by using the RFcircuit 410, or outputs the audio data to the memory 420 for furtherprocessing.

Wi-Fi belongs to a short distance wireless transmission technology. Themobile phone may help, by using the Wi-Fi module 470, a user to receiveand send an email, browse a web page, access stream media, and the like.This provides wireless broadband Internet access for the user. AlthoughFIG. 11 shows the Wi-Fi module 470, it may be understood that the Wi-Fimodule 470 is not a necessary component of the mobile phone, and theWi-Fi module may be omitted as required provided that the scope of theessence of the application is not changed.

The processor 480 is a control center of the mobile phone, and isconnected to various parts of the entire mobile phone by using variousinterfaces and lines. By running or executing a software program and/ormodule stored in the memory 420, and invoking data stored in the memory420, the processor 480 executes various functions of the mobile phoneand performs data processing, thereby monitoring the entire mobilephone. Optionally, the processor 480 may include one or more processingunits. Optionally, the processor 480 may integrate an applicationprocessor and a modem processor. The application processor mainlyprocesses an operating system, a user interface, an application program,and the like. The modem processor mainly processes wirelesscommunication. It may be understood that the foregoing modem may eithernot be integrated into the processor 480.

The mobile phone further includes the power supply 490 (such as abattery) for supplying power to the components. Optionally, the powersupply may be logically connected to the processor 480 by using a powermanagement system, thereby implementing functions such as charging,discharging and power consumption management by using the powermanagement system.

Although not shown in the figure, the mobile phone may further include acamera, a Bluetooth module, and the like, which are not furtherdescribed herein.

In one embodiment of the present disclosure, the processor 480 includedin the terminal further has the following functions: acquiring a firstimage of a target object; extracting coordinate information of a first2D key point corresponding to the target object from the first image;acquiring a second image of the target object; acquiring coordinateinformation of a second 2D key point in the second image according tothe coordinate information of the first 2D key point, the first 2D keypoint and the second 2D key point being the same key point in the targetobject; acquiring coordinate information of a 3D key point correspondingto the target object, the 3D key point being a key point obtained aftertriangulation processing; and determining camera posture information ofthe second image according to the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key point, andthe coordinate information of the 3D key point, the camera postureinformation being used for determining a position of the target objectin the second image.

Optionally, the processor 480 is further configured to perform thefollowing operations: determine whether a number of second 2D key pointsin the second image reaches a threshold; and when the number of thesecond 2D key points reaches the threshold, trigger the operation ofacquiring coordinate information of a second 2D key point in the secondimage according to the coordinate information of the first 2D key point.

Optionally, the processor 480 is specifically configured to perform thefollowing operations: determining the first 2D key point from the firstimage; determining the second 2D key point corresponding to the first 2Dkey point from the first image; and determining the coordinateinformation of the second 2D key point according to the coordinateinformation of the first 2D key point and displacement information,where the displacement information is a distance from the first 2D keypoint to the second 2D key point.

Optionally, the processor 480 is further configured to perform thefollowing operations: when the number of the second 2D key points doesnot reach the threshold, acquiring n key points from the first image, toenable a sum of the number of the second 2D key points and a number ofthe n key points to reach the threshold, where n is a positive integer,the n key points are acquired from a target area corresponding to thefirst image, the first image includes a plurality of equally dividedareas, and a number of the key points in the target area is less than apreset key point threshold; and acquiring coordinate information of then key points and coordinate information of the second 2D key points inthe second image according to the coordinate information of the first 2Dkey point and coordinate information of the n key points.

Optionally, the processor 480 is specifically configured to perform thefollowing operations: determining the first 2D key point and the n keypoints from the first image; determining the second 2D key pointscorresponding to the first 2D key point and the n key points from thefirst image; and determining the coordinate information of the n keypoints and the coordinate information of the second 2D key points in thesecond image according to the coordinate information of the first 2D keypoint, the coordinate information of the n key points, and displacementinformation, where the displacement information includes a distance fromthe first 2D key point to the second 2D key point and distances from then key points to the second 2D key points.

Optionally, the processor 480 is further configured to perform thefollowing operations: performing triangulation processing on thecoordinate information of the first 2D key point, the coordinateinformation of the second 2D key points, camera posture information ofthe first image, and the camera posture information of the second image,and obtaining a 3D key point to be selected and coordinate informationof the 3D key point to be selected; determining an observation angleaccording to the 3D key point to be selected, where the observationangle is an angle between a connecting line between a center point inthe first image and the 3D key point to be selected and a connectingline between a center point in the second image and the 3D key point tobe selected; and when the observation angle is greater than a presetangle, determining the coordinate information of the 3D key point to beselected as coordinate information of a target 3D key point.

Optionally, the processor 480 is further configured to perform thefollowing operations: generating a tracking record, where the trackingrecord is used for recording coordinate information of 2D key points ina plurality of images, the first image in the tracking record is thefirst image, the last image in the tracking record is the second image,and at least one image is included between the first image and thesecond image.

Optionally, the processor 480 is specifically configured to perform thefollowing operations: acquiring coordinate information of a third 2D keypoint of the target object and coordinate information of a fourth 2D keypoint of the target object according to the tracking record, where thethird 2D key point belongs to a third image, the fourth 2D key pointbelongs to a fourth image, and the third image and the fourth image areboth located between the first image and the second image; performingtriangulation processing on the coordinate information of the first 2Dkey point and the coordinate information of the third 2D key point, andobtaining coordinate information of a first 3D key point; performingtriangulation processing on the coordinate information of the third 2Dkey point and the coordinate information of the fourth 2D key point, andobtaining coordinate information of a second 3D key point; determining afirst observation angle according to the coordinate information of thefirst 3D key point, and determining a second observation angle accordingto the coordinate information of the second 3D key point, where thefirst observation angle is an angle between a connecting line between acenter point in the first image and the first 3D key point and aconnecting line between a center point in the third image and the first3D key point, and the second observation angle is an angle between aconnecting line between a center point in the second image and thesecond 3D key point and a connecting line between a center point in thethird image and the second 3D key point; when the first observationangle is greater than the second observation angle, determining thecoordinate information of the first 3D key point as the coordinateinformation of the 3D key point of the target object; and when thesecond observation angle is greater than the first observation angle,determining the coordinate information of the second 3D key point as thecoordinate information of the 3D key point of the target object.

Optionally, the processor 480 is specifically configured to perform thefollowing operations: acquiring a 3D-2D projection error according tothe coordinate information of the 3D key point; acquiring a 2D-2Depipolar distance according to the coordinate information of the first2D key point and the coordinate information of the second 2D key point;and calculating a target function according to the 3D-2D projectionerror and the 2D-2D epipolar distance, and finding the minimum of thetarget function, to obtain the camera posture information of the secondimage.

Optionally, the processor 480 is specifically configured to perform thefollowing operations: calculating the 3D-2D projection error by usingthe following formula:

$\begin{matrix}{{A = {\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}}},} & (4)\end{matrix}$

where A denotes the 3D-2D projection error, C_(3D2D) denotes a set of 3Dkey points projected on the second image, u_(i) denotes the coordinateinformation of the second 2D key point, π_(x) denotes a projectionfunction for 3D-2D projection, the projection function and the cameraposture information have an association relationship, and p_(i) denotesthe coordinate information of the 3D key point.

The acquiring a 2D-2D epipolar distance according to the coordinateinformation of the first 2D key point and the coordinate information ofthe second 2D key point includes: calculating the 2D-2D epipolardistance by using the following formula:

$\begin{matrix}{{B = {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}},} & (5)\end{matrix}$

where B denotes the 2D-2D epipolar distance, C_(2D2D) denotes a set ofthe second 2D key points and the first 2D key points, λ denotes a presetequilibrium constant, the preset equilibrium constant is used forkeeping a difference between 2D-2D epipolar distance and the 3D-2Dprojection error within a preset range, u′_(i) denotes the coordinateinformation of the first 2D key point, F_(x) denotes an elementarymatrix between the second image and the first image, and the projectionfunction and the camera posture information have an associationrelationship.

The calculating a target function according to the 3D-2D projectionerror and the 2D-2D epipolar distance may include: calculating thetarget function by using the following formula:

$\begin{matrix}{{{E(x)} = {{\sum\limits_{i \in C_{3{D2D}}}{{u_{i} - {\pi_{x}\left( p_{i} \right)}}}_{2}^{2}} + {\lambda{\sum\limits_{i \in C_{2{D2D}}}\frac{\left( {u_{i}^{\prime}F_{x}u_{i}} \right)^{2}}{\left( {F_{x}u_{i}} \right)_{1}^{2} + \left( {F_{x}u_{i}} \right)_{2}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{1}^{2} + \left( {F_{x}u_{i}^{\prime}} \right)_{2}^{2}}}}}},} & (6)\end{matrix}$

where E(x) denotes the target function, and x denotes the camera postureinformation of the second image.

All or some of the foregoing embodiments may be implemented by usingsoftware, hardware, firmware, or any combination thereof. When softwareis used for implementation, implementation may be entirely or partiallyperformed in the form of a computer program product.

The computer program product includes one or more computer instructions.When the computer program instructions are loaded and executed on acomputer, all or some of the processes or functions according to theembodiments of the present disclosure are produced. The computer may bea general-purpose computer, a dedicated computer, a computer network, oranother programmable apparatus. The computer instructions may be storedin a computer-readable storage medium, or transmitted from onecomputer-readable storage medium to another computer-readable storagemedium. For example, the computer instructions may be transmitted fromone website, computer, server, or data center to another website,computer, server, or data center in a wired (for example, a coaxialcable, an optical fiber, or a digital subscriber line (DSL)) or wireless(for example, infrared, radio, or microwave) manner. Thecomputer-readable storage medium may be any usable medium accessible bythe computer, or a data storage device, such as a server or a datacenter, including one or more usable media. The available medium may bea magnetic medium (for example, a floppy disk, a hard disk, or a tape),an optical medium (for example, a digital versatile disc (DVD)), or asemi-conductive medium (for example, a solid state disk (SSD)).

A person skilled in the art may clearly understand that, for simple andclear description, for specific work processes of the foregoingdescribed system, apparatus, and unit, reference may be made tocorresponding process in the foregoing method embodiments, and detailsare not described herein again.

In the embodiments provided in the present disclosure, it is to beunderstood that the disclosed system, apparatus, and method may beimplemented in other manners. For example, the described apparatusembodiments are merely exemplary. For example, the unit division ismerely logical function division and may be other division during actualimplementation. For example, a plurality of units or components may becombined or integrated into another system, or some features may beignored or not performed. In addition, the displayed or discussed mutualcouplings or direct couplings or communication connections may beimplemented through some interfaces. The indirect couplings orcommunication connections between the apparatuses or units may beimplemented in electronic, mechanical, or other forms.

The units described as separate parts may or may not be physicallyseparate, and parts displayed as units may or may not be physical units.That is, they may be located in one position, or may be distributed on aplurality of network units. A part of or all of the units may beselected according to actual needs to achieve the objectives of thesolutions of the embodiments.

In addition, functional units in the embodiments of the presentdisclosure may be integrated into one processing unit, or each of theunits may exist alone physically, or two or more units are integratedinto one unit. The integrated unit may be implemented in a form ofhardware, or may be implemented in a form of a software function unit.

When the integrated unit is implemented in the form of a softwarefunctional unit and sold or used as an independent product, theintegrated unit may be stored in a computer-readable storage medium.Based on such an understanding, the technical solutions of the presentdisclosure essentially, or the part contributing to the related art, orall or some of the technical solutions may be implemented in a form of asoftware product. The computer software product is stored in a storagemedium and includes several instructions for instructing a computerdevice (which may be a PC, a server, a network device, or the like) toperform all or some of the steps of the methods described in theembodiments of the present disclosure. The foregoing storage mediumincludes: any medium that can store program code, such as a USB flashdrive, a removable hard disk, a read-only memory (ROM), a random accessmemory (RAM), a magnetic disk, or an optical disc.

The foregoing embodiments are merely provided for describing thetechnical solutions of the present disclosure, but not intended to limitthe present disclosure. It is to be understood by a person of ordinaryskill in the art that although the present disclosure has been describedin detail with reference to the foregoing embodiments, modifications canbe made to the technical solutions described in the foregoingembodiments, or equivalent replacements can be made to some technicalfeatures in the technical solutions, as long as such modifications orreplacements do not cause the essence of corresponding technicalsolutions to depart from the spirit and scope of the present disclosure.

What is claimed is:
 1. A method for an image capturing device,comprising: acquiring a first image of a target object; extractingcoordinate information of a first two-dimensional (2D) key pointcorresponding to the target object from the first image; acquiring asecond image of the target object; acquiring coordinate information of asecond 2D key point in the second image according to the coordinateinformation of the first 2D key point, the first 2D key point and thesecond 2D key point being the same key point in the target object;acquiring coordinate information of a three-dimensional (3D) key pointcorresponding to the target object, the 3D key point being a key pointobtained after a triangulation operation; determining camera postureinformation of the second image according to the coordinate informationof the first 2D key point, the coordinate information of the second 2Dkey point, the coordinate information of the 3D key point, and a 3D-2Dprojection, the camera posture information being used for determining aposition of the target object in the second image; acquiring additionalimage information; and generating an enhanced image for augmentedreality (AR) display according to the second image, the camera postureinformation of the second image, and the additional image information.2. The method according to claim 1, further comprising: determiningwhether a number of target 2D key points in the second image reaches athreshold, the target 2D key points corresponding to the target objectin the second image; and in response to determining that the number ofthe target 2D key points reaches the threshold, acquiring the coordinateinformation of the second 2D key point in the second image according tothe coordinate information of the first 2D key point.
 3. The methodaccording to claim 1, wherein the acquiring coordinate information of asecond 2D key point in the second image according to the coordinateinformation of the first 2D key point comprises: determining the first2D key point from the first image; determining the second 2D key pointcorresponding to the first 2D key point from the first image; anddetermining the coordinate information of the second 2D key pointaccording to the coordinate information of the first 2D key point anddisplacement information, wherein the displacement information is adistance from the first 2D key point to the second 2D key point.
 4. Themethod according to claim 2, further comprising: when the number of thetarget 2D key points does not reach the threshold, acquiring n first keypoints from the first image, to enable a sum of the number of the target2D key points and a number of the n first key points to reach thethreshold, wherein n is a positive integer, the n first key points areacquired from a target area corresponding to the first image, the firstimage comprises a plurality of equally divided areas, and a number ofthe key points in the target area is less than a preset key pointthreshold; and acquiring coordinate information of second 2D key pointscorresponding to the second image according to the coordinateinformation of the first 2D key point and coordinate information of then first key points, the second 2D key points including the second 2D keypoint and n second key points, and each of the n second key pointshaving a corresponding key point in the n first key point thatrepresents a same key point in the target object.
 5. The methodaccording to claim 4, wherein the acquiring coordinate information ofthe second 2D key points in the second image comprises: determining thefirst 2D key point and the n key points from the first image;determining the second 2D key points corresponding to the first 2D keypoint and the n first key points from the first image; and determiningthe coordinate information of the second 2D key points corresponding tothe second image according to the coordinate information of the first 2Dkey point, the coordinate information of the n first key points, anddisplacement information, wherein the displacement information comprisesa distance from the first 2D key point to the second 2D key point. 6.The method according to claim 1, further comprising: performingtriangulation processing on the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key points,camera posture information of the first image, and the camera postureinformation of the second image, and obtaining a 3D key point to beselected and coordinate information of the 3D key point to be selected;and determining an observation angle according to the 3D key point to beselected, wherein the observation angle is an angle between a connectingline between a center point in the first image and the 3D key point tobe selected and a connecting line between a center point in the secondimage and the 3D key point to be selected; and when the observationangle is greater than a preset angle, determining the coordinateinformation of the 3D key point to be selected as coordinate informationof a target 3D key point.
 7. The method according to claim 1, whereinafter the acquiring coordinate information of a second 2D key point inthe second image according to the coordinate information of the first 2Dkey point, the method further comprises: generating a tracking record,wherein the tracking record is used for recording coordinate informationof 2D key points in a plurality of images, the first image in thetracking record is the first image, the last image in the trackingrecord is the second image, and at least one image is comprised betweenthe first image and the second image.
 8. The method according to claim7, wherein the acquiring coordinate information of a 3D key pointcorresponding to the target object comprises: acquiring coordinateinformation of a third 2D key point of the target object and coordinateinformation of a fourth 2D key point of the target object according tothe tracking record, wherein the third 2D key point belongs to a thirdimage, the fourth 2D key point belongs to a fourth image, and the thirdimage and the fourth image are both located between the first image andthe second image; performing the triangulation operation on thecoordinate information of the first 2D key point and the coordinateinformation of the third 2D key point, and obtaining coordinateinformation of a first 3D key point; performing the triangulationoperation on the coordinate information of the third 2D key point andthe coordinate information of the fourth 2D key point, and obtainingcoordinate information of a second 3D key point; determining a firstobservation angle according to the coordinate information of the first3D key point, and determining a second observation angle according tothe coordinate information of the second 3D key point, wherein the firstobservation angle is an angle between a connecting line between a centerpoint in the first image and the first 3D key point and a connectingline between a center point in the third image and the first 3D keypoint, and the second observation angle is an angle between a connectingline between a center point in the second image and the second 3D keypoint and a connecting line between a center point in the third imageand the second 3D key point; when the first observation angle is greaterthan the second observation angle, determining the coordinateinformation of the first 3D key point as the coordinate information ofthe 3D key point of the target object; and when the second observationangle is greater than the first observation angle, determining thecoordinate information of the second 3D key point as the coordinateinformation of the 3D key point of the target object.
 9. The methodaccording to claim 1, wherein the determining camera posture informationof the second image according to the coordinate information of the first2D key point, the coordinate information of the second 2D key point, andthe coordinate information of the 3D key point comprises: acquiring a3D-2D projection error according to the coordinate information of the 3Dkey point; acquiring a 2D-2D epipolar distance according to thecoordinate information of the first 2D key point and the coordinateinformation of the second 2D key point; and calculating a targetfunction according to the 3D-2D projection error and the 2D-2D epipolardistance, and finding the minimum of the target function, to obtain thecamera posture information of the second image.
 10. The method accordingto claim 9, wherein the acquiring a 3D-2D projection error according tothe coordinate information of the 3D key point and the coordinateinformation of the second 2D key point comprises: calculating the 3D-2Dprojection error according to a set of 3D key points projected on thesecond image, the coordinate information of the second 2D key point, aprojection function for 3D-2D projection with regard to the coordinateinformation of the 3D key point, the projection function and the cameraposture information have an association relationship; the acquiring a2D-2D epipolar distance according to the coordinate information of thefirst 2D key point and the coordinate information of the second 2D keypoint comprises: calculating the 2D-2D epipolar distance according to aset of the second 2D key points and the first 2D key points, a presetequilibrium constant used for keeping a difference between 2D-2Depipolar distance and the 3D-2D projection error within a preset range,the coordinate information of the first 2D key point, an elementarymatrix between the second image and the first image; and the calculatinga target function according to the 3D-2D projection error and the 2D-2Depipolar distance comprises: calculating the target function accordingto a sum of the 3D-2D epipolar distance and the 2D-2D epipolar distance.11. The method according to claim 1, wherein: the additional imageinformation is an image of a photographed real scene; and generating theenhanced image includes: combining and displaying the image of thephotographed real scene, the second image, and a movable contentcorresponding to the second image.
 12. An image capturing device,comprising: a memory being configured to store a computer program; aprocessor connected to the memory and being configured to execute thecomputer program in the memory to perform: acquiring a first image of atarget object; extracting coordinate information of a firsttwo-dimensional (2D) key point corresponding to the target object fromthe first image; acquiring a second image of the target object;acquiring coordinate information of a second 2D key point in the secondimage according to the coordinate information of the first 2D key point,the first 2D key point and the second 2D key point being the same keypoint in the target object; acquiring coordinate information of athree-dimensional (3D) key point corresponding to the target object, the3D key point being a key point obtained after a triangulation operation;determining camera posture information of the second image according tothe coordinate information of the first 2D key point, the coordinateinformation of the second 2D key point, the coordinate information ofthe 3D key point, and a 3D-2D projection, the camera posture informationbeing used for determining a position of the target object in the secondimage; acquiring additional image information; and generating anenhanced image for augmented reality (AR) display according to thesecond image, the camera posture information of the second image, andthe additional image information.
 13. The image capturing deviceaccording to claim 12, wherein the processor is further configured toperform: performing the triangulation operation on the coordinateinformation of the first 2D key point, the coordinate information of thesecond 2D key points, camera posture information of the first image, andthe camera posture information of the second image, and obtaining a 3Dkey point to be selected and coordinate information of the 3D key pointto be selected; determining an observation angle according to the 3D keypoint to be selected, wherein the observation angle is an angle betweena connecting line between a center point in the first image and the 3Dkey point to be selected and a connecting line between a center point inthe second image and the 3D key point to be selected; and when theobservation angle is greater than a preset angle, determining thecoordinate information of the 3D key point to be selected as coordinateinformation of a target 3D key point.
 14. The image capturing deviceaccording to claim 12, wherein the processor is further configured toperform: determining whether a number of target 2D key points in thesecond image reaches a threshold, the target 2D key points correspondingto the target object in the second image; and in response to determiningthat the number of the target 2D key points reaches the threshold,acquiring the coordinate information of the second 2D key point in thesecond image according to the coordinate information of the first 2D keypoint.
 15. The image capturing device according to claim 12, wherein theacquiring coordinate information of a second 2D key point in the secondimage according to the coordinate information of the first 2D key pointcomprises: determining the first 2D key point from the first image;determining the second 2D key point corresponding to the first 2D keypoint from the first image; and determining the coordinate informationof the second 2D key point according to the coordinate information ofthe first 2D key point and displacement information, wherein thedisplacement information is a distance from the first 2D key point tothe second 2D key point.
 16. The image capturing device according toclaim 14, wherein the processor is further configured to perform: whenthe number of the target 2D key points does not reach the threshold,acquiring n first key points from the first image, to enable a sum ofthe number of the target 2D key points and a number of the n first keypoints to reach the threshold, wherein n is a positive integer, the nfirst key points are acquired from a target area corresponding to thefirst image, the first image comprises a plurality of equally dividedareas, and a number of the key points in the target area is less than apreset key point threshold; and acquiring coordinate information ofsecond 2D key points corresponding to the second image according to thecoordinate information of the first 2D key point and coordinateinformation of the n first key points, the second 2D key pointsincluding the second 2D key point and n second key points, and each ofthe n second key points having a corresponding key point in the n firstkey point that represents a same key point in the target object.
 17. Theimage capturing device according to claim 16, wherein the acquiringcoordinate information of the second 2D key points in the second imagecomprises: determining the first 2D key point and the n key points fromthe first image; determining the second 2D key points corresponding tothe first 2D key point and the n first key points from the first image;and determining the coordinate information of the second 2D key pointscorresponding to the second image according to the coordinateinformation of the first 2D key point, the coordinate information of then first key points, and displacement information, wherein thedisplacement information comprises a distance from the first 2D keypoint to the second 2D key point.
 18. A non-transitory computer-readablestorage medium storing computer program instructions executable by atleast one processor to perform: acquiring a first image of a targetobject; extracting coordinate information of a first two-dimensional(2D) key point corresponding to the target object from the first image;acquiring a second image of the target object; acquiring coordinateinformation of a second 2D key point in the second image according tothe coordinate information of the first 2D key point, the first 2D keypoint and the second 2D key point being the same key point in the targetobject; acquiring coordinate information of a three-dimensional (3D) keypoint corresponding to the target object, the 3D key point being a keypoint obtained after a triangulation operation; determining cameraposture information of the second image according to the coordinateinformation of the first 2D key point, the coordinate information of thesecond 2D key point, the coordinate information of the 3D key point, anda 3D-2D projection, the camera posture information being used fordetermining a position of the target object in the second image;acquiring additional image information; and generating an enhanced imagefor augmented reality (AR) display according to the second image, thecamera posture information of the second image, and the additional imageinformation.
 19. The non-transitory computer-readable storage mediumaccording to claim 18, wherein, the additional image information is animage of a photographed real scene; and generating the enhanced imageincludes: combining and displaying the image of the photographed realscene, the second image, and a movable content corresponding to thesecond image
 20. The non-transitory computer-readable storage mediumaccording to claim 18, wherein the computer program instructions furthercause the at least one processor to perform: performing thetriangulation operation on the coordinate information of the first 2Dkey point, the coordinate information of the second 2D key points,camera posture information of the first image, and the camera postureinformation of the second image, and obtaining a 3D key point to beselected and coordinate information of the 3D key point to be selected;determining an observation angle according to the 3D key point to beselected, wherein the observation angle is an angle between a connectingline between a center point in the first image and the 3D key point tobe selected and a connecting line between a center point in the secondimage and the 3D key point to be selected; and when the observationangle is greater than a preset angle, determining the coordinateinformation of the 3D key point to be selected as coordinate informationof a target 3D key point.